Scientific research has demonstrated that memory and consciousness reside in the organization of neurons, their interconnections, charges on the cell surfaces, intracellular and extracellular proteins and other molecules, and other factors. Although the ability to understand the molecular and cellular mechanism of memory and consciousness, the sequences, the charges, and how they relate to memory and consciousness is in very early development, the capability to record and analyze these data in detail exists now.
Great strides have been made in the area of functional and structural imaging of the human brain. The ability to interpret and correlate brain imaging information and the stimuli that result in memories, thoughts and concepts is ready for development. The technology for imaging the macroscopic, microscopic and molecular structure of individual human brains, through techniques such as Computed Axial Tomography (CT or CAT), functional Magnetic Resonance Imaging (fMRI), Positron Emission Tomography (PET), electroencephalography (EEG), and Magnetoencephalography (MEG) (collectively referred hereafter as neuroimaging technologies) is continually improving. Recent progress in improved imaging resolution and advancing computational analysis of the data has led to evolving understanding of where thoughts are encoded in the brain.
Neuroanatomy Using Imaging
Kim et al. (2003) used Blood Oxygenation Level-Dependent (BOLD)-based functional MRI (fMRI) and diffusion tensor imaging (DTI) on the cat visual cortex to allow three-dimensional fiber reconstruction, to construct a map of the axonal circuitry underlying visual information processing. Rueckert et al. (2003) used the concept of statistical deformation modeling to construct average models of neuroanatomy and variability of 25 different human subjects. With newer MRI machines with stronger (3 Tesla and above) magnetic fields and improved software, cellular resolution is now being attempted.
Brain Organization and Imaging
Zeineh et al. (2003) used high-resolution FMRI to study the process of encoding and retrieval of memories of names associated with faces, in the medial temporal lobe (MTL) of the human brain and its subregions. The cornu ammonis and the dentate gyrus regions of the brain were active relative to baseline only during encoding, and this activity decreased as associations were learned. Activity in the subiculum region showed the same temporal decline, but primarily during retrieval. Zeineh and colleagues evaluated changes in the blood oxygen level-dependent (BOLD) response, reflecting neural activity, within different substructures of the MTL, as subjects progressively learned new associations. These researchers developed techniques to acquire high-resolution structural (0.4 by 0.4 mm) and functional (1.6 by 1.6 mm) MRI data and to localize functional activity precisely within the substructures of the hippocampus. Similar work, with modifications in technique, particularly stimulus and matching activity, forms one of the basic techniques of patent application #10218 for cognitive engineering. Zieneh et al. manipulated the imaging data to mathematically represent an “unfolding” of the hippocampal cortex, which revealed the entirety of each hippocampal subregion (within the resolution restrictions of the equipment) and adjacent neocortical regions (parahippocampal, entorhinal, perirhinal, and fusiform) in a single plane, or “flat map” representation. Boundaries were demarcated between the architectonic subregions on the high-resolution structural MR images. The white matter and CSF throughout the MTL were segmented and separated out, retaining only the gray matter sheath. Gray matter was then computationally extracted and flattened (similar to flattening the globe into a flat map of the world) to project the demarcated boundaries to produce unfolded flattened maps of the hippocampus.
Using these techniques, Zeineh et al. studied ten subjects, who were scanned while they performed a face-name association task in which a series of unfamiliar (could be familiar) faces were paired with names. Computational warping techniques transformed an individual subject's hippocampal maps to the flat hippocampal template space. The same transformation parameters were then applied to the coregistered functional MRI scans, which delivered high-resolution fMRI data in a standardized flat space. This procedure enabled measuring activity over time in each subregion and to perform powerful group statistics across subjects. Similarly, cognitive engineering will create a data model of a concept, and of specific visual objects, such as a face.
Zeineh and colleagues were able to show a strong, parametric correlation between activity in specific brain areas and the storage of new associations. As the number of new associations learned decreased from block to block, activity in these regions fell in parallel. They also found a similarly strong relationship between activity in the subiculum and retrieval of newly learned associations.
Because subjects vary in the anatomy of their MTLs, Zeineh et al. constructed a template representing the typical anatomy of the subject population by averaging together the individual demarcation boundaries across subjects. This is somewhat analogous to the work of Rueckert (automatic construction of three-dimensional statistical deformation models (SDM) of the brain using nonrigid registration). Using a group-averaged incremental performance curve, the researchers regressed MR signal intensity in each pixel and each subject with two waveforms reflecting either performance during learning or performance during retrieval, and then statistically tested whether the slope of each regression for a given pixel was on average different from zero.
In summary, Zeineh et al. identified mnemonic properties of different subregions within the hippocampal circuitry as human subjects learned to associate names with faces. The cornu ammonis (CA) .elds 2 and 3 and the dentate gyrus were active relative to baseline only during encoding, and this activity decreased as associations were learned. Activity in the subiculum showed the same temporal decline, but primarily during retrieval.
Ishai and Ungerleider (1999) identified, using FMRI, three bilateral regions in the ventral temporal cortex that responded preferentially to faces, houses, and chairs. In a follow-up report (Ishai 2000) they demonstrated differential patterns of activation, similar to those seen in the ventral temporal cortex, in the bilateral regions of the ventral occipital cortex. They also found category-related responses in the dorsal occipital cortex and in the superior temporal sulcus. Moreover, rather than activating discrete, segregated areas, each category was associated with its own differential pattern of response across a broad expanse of cortex.
The distributed patterns of response were similar across tasks (passive viewing, delayed matching) and presentation formats (photographs, line drawings). Ishai et al. (2000) proposed that the representation of objects in the ventral visual pathway, including both occipital and temporal regions, is not restricted to small, highly selective patches of cortex but, instead, is a distributed representation of information about object form. Within this distributed system, the representation of faces appears to be less extensive as compared to the representations of non-face objects.
Koechlin et al. (2003) showed that the lateral pre-frontal cortex (PFC) of the brain is organized as a cascade of executive (controlling) processes from premotor to anterior PFC regions. These processes control behavior according to stimuli, the present perceptual context, and the temporal episode in which stimuli occur, respectively. Koechlin et al.'s results support a unified modular model of cognitive control that describes the overall functional organization of the human lateral PFC and has basic methodological and theoretical implications.
Fan et al. (2003) studied whether source information, item information, or both are required at the time of memory retrieval. Two sources were used in a factorial design in which the main effect of source and item retrieval, along with their interaction, could be measured by fMRI activations. They found that when source information was required at retrieval, the left frontal lobe showed significant activation but not when item retrieval was required. Activation of the hippocampal section of the brain showed no difference between source and item retrieval. Fan et al.'s data supports a larger role for the frontal lobes in encoding and retrieval of source information.
Nielson et al. (2004), utilizing statistical data mining of a neuroimaging database, located associations between various words/text and brain locations. This provided an understanding of how the brain associates words indicative of cognitive function.
It appears that in all of the studies to date, neuroimaging researchers have mapped gross brain functional activation with various macroscopic regions of the brain. There has not been an attempt to understand how individual brain imaging is directly linked to the concepts that form the basis of thought. This is the principal area in which the present (Cognitive Engineering® 10218) patent application differs.
Correlations Between Functioning of Various Brain Regions
Givens et al. (1999) reviewed their and other data using EEG to study higher brain function. They emphasized the ability of more modern EEG studies to complement functional neuroimaging techniques. The current invention (Cognitive Engineering® 10218) may utilize multiple simultaneous neuroimaging techniques, including supplementation by EEG, during the construction of some data sets.
Suppes et al. (1998) studied the ability of recordings of electrical and magnetic brain waves of two subjects to recognize which one of twelve sentences or seven words auditorily presented was processed. The analysis consisted of averaging over trials to create prototypes and test samples, to each of which a Fourier transform was applied, followed by filtering and an inverse transformation to the time domain. The filters used were optimal predictive filters, selected for each subject. A still further improvement was obtained by taking differences between recordings of two electrodes to obtain bipolar pairs that then were used for the same analysis. Recognition rates, based on a least-squares criterion, varied, but the best were above 90%. The first words of prototypes of sentences also were cut and pasted to test, at least partially, the invariance of a word's brain wave in different sentence contexts. The best result was above 80% correct recognition. Test samples made up only of individual trials also were analyzed. The best result was 134 correct of 288 (47%), compared to the expected recognition number by chance (24, or 8.3%).
Hartley and Speer (2000) reviewed functional neuroimaging data of memory, and discussed progress in understanding memory systems. Rugg et al. (2002) reviewed FMRI use to study episodic memory in humans. The data they review, while impressive, does not localize below general brain region areas of activation.
Fujii et al. (2002) used PET to image normal volunteers engaged in deep (semantic) or shallow (phonological) processing of new or repeated words. Their results showed that deep processing, compared with shallow processing, resulted in significantly better recognition performance and that this effect was associated with activation of various brain areas. Regions directly relevant to episodic memory encoding were located in the anterior part of the parahippocampal gyrus, inferior frontal gyrus, supramarginal gyrus, anterior cingulate gyrus, and medial frontal lobe in the left hemisphere. The authors concluded that several regions, including the medial temporal lobe, play a role in episodic memory encoding.
Zhang et al. (2003) studied the involvement of frontal cortex in accessing and evaluating information in working memory using a variant of a Sternberg paradigm and comparing brain activations between positive and negative responses (known to differentially tax access/evaluation processes). Test subjects remembered two trigrams in each trial and were then cued to discard one of them and maintain the other one as the target set. After a delay, a probe letter was presented and participants made decisions about whether or not it was in the target set. Several frontal areas—anterior cingulate (BA32), middle frontal gyrus (bilateral BA9, right BA10, and right BA46), and left inferior frontal gyrus (BA44/45)—showed increased activity when participants made correct negative responses relative to when they made correct positive responses. No areas activated significantly more for the positive responses than for the negative responses. The authors suggested that the multiple frontal areas involved in the test phase of this task may reflect several component processes that underlie more general frontal functions.
Schmithorst and Holland (2004) used fMRI to study neural correlates of the link between formal musical training and mathematics performance in normal adults. Musical training was associated with increased activation in the left fusiform gyrus and prefrontal cortex areas of the brain, and decreased activation in visual association areas and the left inferior parietal lobule of the brain during a mathematical task. The authors hypothesized that the correlation between musical training and math proficiency may be associated with improved working memory performance and an increased abstract representation of numerical quantities.
Lanius et al. (2004) used both 4-T fMRI and functional connectivity analyses to assess interregional brain activity correlations during the recall of traumatic memories in traumatized subjects with and without posttraumatic stress disorder (PTSD). Comparison of connectivity maps at the right anterior cingulate gyrus brain region for the two groups showed that the subjects without PTSD had greater correlation than the PTSD subjects in the following brain areas: left superior frontal gyrus (Brodmann's area 9), left anterior cingulate gyrus (Brodmann's area 32), left striatum (caudate), left parietal lobe (Brodmann's areas 40 and 43), and left insula (Brodmann's area 13). In contrast, the PTSD subjects showed greater correlation than the subjects without PTSD in the right posterior cingulate gyrus (Brodmann's area 29), right caudate, right parietal lobe (Brodmann's areas 7 and 40), and right occipital lobe (Brodmann's area 19). The authors concluded that the differences in brain connectivity between PTSD and comparison subjects may account for the nonverbal nature of traumatic memory recall in PTSD subjects, compared to a more verbal pattern of traumatic memory recall in comparison subjects.
Visual Recognition in the Brain
Na et al. (2000) used fMRI to image working memory in humans. Like all studies to date, they were able to determine gross brain areas of activation, but were limited on their resolution. Na et al. assessed activated brain areas during stimulation tasks (item recognition), followed by an activation period. The prefrontal cortex and secondary visual cortex were activated bilaterally by both verbal and visual working memory tasks, and the patterns of activated signals were similar in both tasks. Bilateral prefrontal and superior parietal cortices activated by the visual working memory task may be related to the visual maintenance of objects, representing visual working memory.
Their activation map images of the upper level of the brain showed neither activated signals in the supramarginal gyrus nor lateralization of activated signals in the frontal and parietal lobes. Map image of the middle level of the brain showed no activated signals in the left inferior frontal or temporal gyrus. An activated signal in the prefrontal cortex corresponded to the signal activated during the verbal working memory task. Map image of the lower level of the brain showed bilateral activated signals similar to those seen during the verbal working memory task in the right and left occipital cortices and posterior fusiform gyri.
Zhang et al. (2003) compared brain activations between positive and negative responses (known to differentially tax access/evaluation processes) to investigate the involvement of frontal cortex in accessing and evaluating information in working memory. Participants remembered two trigrams in each trial and were then cued to discard one of them and maintain the other one as the target set. After a delay, a probe letter was presented and participants made decisions about whether or not it was in the target set. Several frontal areas—anterior cingulate (BA32), middle frontal gyrus (bilateral BA9, right BA10, and right BA46), and left inferior frontal gyrus (BA44/45)—showed increased activity when participants made correct negative responses relative to when they made correct positive responses.
Visual Imaging Using Techniques Other than FMRI
Blaizot et al. (2000) used PET data to map the visual recognition memory network in the baboon. Using computerized matching to sample visuomotor control tasks, they matched PET data to that obtained by anatomic MRI images. They found that foci of significant activation were distributed along the following brain areas: ventral occipitotemporal pathway, inferomedial temporal lobe, and orbitofrontal cortex, consistent with activation studies in healthy humans.
Additional Studies Indicating Location of Active Memory and Thought
In addition to the work of Ishai and of Zenich, referred to above, there are a number of other studies supporting the accessibility of neural data for interpretation of the actual thought processes occurring. Anderson et al. were able to record signals from neurons in monkeys and showed how they were coding for movement, an important step towards creating better prosthetic devices for paralyzed people. The decoded signals enabled the researchers to predict the monkeys' arm movements in tasks in which they thought about reaching for an item without actually doing so. Further, their research suggests that other types of cognitive signals can be decoded from patients.
Past studies on monkeys have shown that information from neurons coding movement instructions can be used to control prosthetic devices. For example, Rhesus monkeys could be taught to control and assimilate a robot arm using signals from their brain. To achieve this, researchers implanted an array of microelectrodes into the frontal and parietal lobes—areas of the brain involved in producing multiple output commands to control complex muscle movements. The faint brain signals from the electrodes were detected and analyzed by a computer system to recognize patterns of signals that represent particular movements by an animal's arm. These signals were translated into similar movements of a robotic arm.
Andersen and colleagues implanted in monkeys arrays of electrodes into areas of the brain that encode the goals of reaching movements rather than controlling movement itself. While the monkeys waited for a cue that told them to reach for an icon flashing on a screen, a computer program interpreted the brain signals recorded by the electrodes. Once the “neuronal code” was cracked, the researchers used the program to decipher the direction that the monkeys were planning to reach for during trials in which they thought about reaching but didn't actually do so. When monkeys remained still while having thoughts that were consistent with requested movements, they received a reward.
At first, the program had trouble matching the monkeys' intentions to the icon's position much more often than chance. As the monkeys practiced thinking about reaching, however, their neural signals became stronger, enabling the program to decode the correct direction more frequently. Eventually, the program could predict the intended direction of the monkeys' reach as much as 67% of the time. When the monkeys knew that accurately thinking about the requested movement would yield a preferred reward, the computer's ability to predict direction improved by as much as 21%. For instance, recording thoughts from speech areas could alleviate the use by those unable to speak (stroke, other neurologic diseases) of more cumbersome letter boards and time-consuming spelling programs, or recordings from emotion centers could provide an online indication of a patient's emotional state.
The overall significance of these studies is that changes in the brain occur during active memory, and these changes can be observed using existing neuroimaging technologies.
Thus, there exists a need to translate neurologic changes occurring upon sensory or physical stimulation into a decipherable code allowing one to effectively read neurologic activity of an individual.